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Abstract 

Grid computing aims to connect large numbers of ge- 
ographically and organizationally distributed resources to 
increase computational power, resource utilization, and re- 
source accessibility. In order to effectively utilize grids, 
users need to be connected to the best available resources 
at any given time. As grids are in constant flux, users cannot 
be expected to keep up with the configuration and status of 
the grid, thus they must be provided with automatic resource 
brokering for selecting and ranking resources meeting con- 
straints and preferences they specify. This paper presents a 
new OGSl-compliant resource selection and ranking frame- 
work called Surfer that has been implemented as part of 
NASA’s Information Power Grid (IPG) project. Surfer is 
highly extensible and may be integrated into any grid en- 
vironment by adding information providers knowledgeable 
about that environment. 


1. Introduction 

Grid computing [4] aims to connect large numbers of ge- 
ographically and organizationally distributed resources to 
increase computational power, resource utilization, and re- 
source accessibility. In order to effectively utilize grids, 
users need to be connected to the best available resources 
at any given time. In small, single organization grids, users 
may be able to adequately select resources for simple re- 
quests on their own. Even small grids, however, are in 
constant flux with resource availability changing minute by 
minute due to user activity and system failures. Large, 
multi-organization grids are much more chaotic where re- 
source types, connectivity, support levels, software environ- 
ment, availability, etc. may vary greatly with a greater prob- 
ability of resource failures. In such an environment, users 
cannot possibly keep up with the configuration and status of 
the grid nor potentially even which resources they may have 


access to, thus they will tend to choose only those resources 
they are directly familiar with. This leads to imbalanced re- 
source utilization, which results in longer wait times and a 
decrease in productivity. To maximize the benefits of grid 
computing, users must be provided with automatic resource 
brokering for selecting and ranking resources meeting con- 
straints and preferences they specify. 

Although resource brokering is a fundamental service 
that is vital to the usability of every grid environment, it is 
difficult to provide a general purpose solution for all such 
environments since each one has its own idiosyncrasies 
such as job models, resource types, and sources of infor- 
mation. Typical resource brokers are tightly entangled with 
these idiosyncrasies, eliminating the possibility of reuse 
across the grid community. By removing any dependence 
on such idiosyncrasies, and specifically, decoupling the ac- 
cess to resource information from its utilization in the re- 
source selection process, it becomes possible to provide a 
general framework upon which resource brokers for any en- 
vironment can be built. The key features required in such a 
framework include an expressive and extensible constraint 
language, easy integration of new resource types and infor- 
mation sources, and accommodating preexisting informa- 
tion retrieval optimizations. 

This paper presents Surfer, the Selection and Ranking 
Framework for Extracting Resources. The job of Surfer is 
to surf the pool of potential grid resources and extract the 
highest ranked resources meeting user specified constraints 
and preferences. Surfer has no built-in bias towards any 
job model or selection policy, thus is suitable for inclusion 
in any grid environment by adding information providers 
knowledgeable about that environment. These providers de- 
termine the types of resources that are selectable and sup- 
ply functions constrainable in resource requests. Provider 
functions are allowed to have arbitrary argument and re- 
turn types or may be macros to be expanded before pro- 
cessing. Function definitions may hide arbitrarily complex 
back-end information retrieval that may be optimized as de- 
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sired. Surfer has been implemented as an OGSI-compliant 
grid service that can also be invoked directly through its 
Java APIs. 

Surfer is part of NASA’s Information Power Grid (IPG) 
project [7]. The goal of the IPG is to develop new tech- 
nologies to facilitate the use of the grid and enable scien- 
tific discovery. Several prototype services have been imple- 
mented including an execution service for submitting and 
managing jobs, a prediction service [18] for estimating ex- 
ecution, wait, and transfer times, the Cardea [10] service 
for dynamic resource access control, the Naturalization Ser- 
vice [8] for automatically establishing the execution envi- 
ronment for user applications, and the Surfer framework, 
which is the subject of this paper. 

This paper is organized as follows. Section 2 presents 
related work. Section 3 describes the selection and ranking 
framework. Section 4 describes a prototype resource bro- 
ker implemented for the IPG using this framework. Finally, 
section 5 presents conclusions and future work. 

2. Related Work 

The most well known resource selection framework is 
the Condor matchmaker [15]. In this framework, providers 
and consumers describe their properties and requirements as 
classified advertisements (classads), which are pushed to a 
central matchmaker that does the matching. Each classad is 
a mapping between attributes and values that may addition- 
ally contain a constraint, which describes the requirements 
that must be met by any matching classads, and a ranking 
function, which describes the order of preference when sev- 
eral classads satisfy the constraint. 

The original matchmaking framework only allows a re- 
quest to be matched with a single classad, but has been ex- 
tended in several ways to overcome this limitation. In [12], 
a request may be matched to a set of classads of the same 
type, where requests may contain aggregate requirements 
that all classads of the set must satisfy. The aggregation 
functions include min, max, and sum. In [16], classads have 
been extended by allowing them to contain multiple ports, 
each of which may be matched with other classads of spe- 
cific types and characteristics. Thus, a single request may 
be matched with multiple heterogeneous classads. 

For matchmaking schemes to work, all parties must uti- 
lize the same vocabulary of attribute names and values. To 
facilitate interoperability between disparate vocabularies, 
[19] extends matchmaking with ontologies, which allows 
conditions to be defined under which differing attribute val- 
ues may still be matched (e.g. “Linux” and “FreeBSD” 
match “Unix”). This approach, however, does not yet sup- 
port multi-classad matching. RedLine [11] incorporates all 
three classad extensions by allowing heterogeneous classad 
sets, aggregate set requirements, and ontological vocabular- 


ies. Requests are specified as a set of constraints, which are 
solved as a constraint satisfaction problem. 

All of the matchmaking models suffer from the same 
limitation. Namely, they rely on a push-based model of in- 
formation, where every possible attribute of interest must be 
computed a priori and stored within a classad to be utilized 
during selection. Many attributes critical to resource selec- 
tion, however, are too dynamic or complex to precompute 
and/or too large to store temporarily. Examples include ac- 
cess control, which may be completely dynamic based on 
a resource’s current state and the user’s grid identity as in 
[10] and network bandwidth, which may include measure- 
ments between a fuiiy connected network of thousands of 
resources. In general, a more flexible pull-based model is 
needed to utilize such information, which can be computed 
on demand to control size and complexity. 

A variety of resource brokers are available with their 
respective grid environments/scheduling systems. Exam- 
ples include resource brokers for Legion [2], European Data 
Grid [9], ALICE Environment [17], UNICORE [13], Fraun- 
hofer Resource Grid [6], Nimrod/G [1], and GridLab [14]. 
In general, these brokers were designed for their specific 
grid environment, thus are not easily extensible, nor are eas- 
ily incorporated into other environments. In some cases, 
they are dependent on a specific job model or job submis- 
sion functionality. In other cases, they are dependent on 
specific information sources that may not be available else- 
where. Finally, many of them have built-in selection poli- 
cies such as a specific load-balancing scheme that may be 
unsuitable and/or undesirable in other environments. 

3. Surfer Framework 

Surfer is a framework for the selection and ranking of 
resources, where a resource is considered to be any entity 
that may require selecting such as computers, storage sys- 
tems, software, data, etc. Figure 1 shows the architecture 
of Surfer, which is described in greater detail in section 3.5. 
Processing begins when a user or client application makes 
a resource selection request. The request is rewritten from 
a boolean expression to a set expression utilizing function 
calls and queries to providers that supply information. The 
rewritten request is then evaluated into an actual set of re- 
source sets, which is finally ranked and returned to the user. 
The following sections describe this process in detail. 

3.1. Requests 

A Surfer request consists of a set of individual resource 
requests, a global constraint, a global ranking function, and 
a number of resource sets to return. Each individual re- 
quest consists of an identifier, a resource type, a local con- 
straint, and a local ranking function. The global constraint 
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Figure 1 . Surfer architecture 


describes the requirements that must be met by the complete 
set of resources while each local constraint describes the re- 
quirements that must be met by any resources selected for 
that particular resource request. Similarly, the global rank- 
ing function describes how complete resource sets should 
be ordered while each local ranking function describes how 
individual resources should be ordered. 

Figure 2 shows an example request for a storage resource 
with more than 20 GB of free disk space and a compute 
resource running IRIX64 with at least 128 free CPUs and 
more than 10 GB of free physical memory, where the host 
name of each must be the same. The most preferable stor- 
age resources have the most free disk space while the most 
preferable compute resources have the most free CPUs. The 
final set of resource pairs must be ordered by the sum of the 
local ranking functions after the compute resource’s rank 
has been scaled by a factor of 100. Note that this figure is 
only a text representation of a request object and does not 
depict a specific syntax for making requests with the excep- 
tion of the constraints and the ranking functions. 


Global: 

Constraint: 

Scl.host = Ssl.host 
Ranking: 

100 * $cl ranking + 
$s l.ranking 
Resource: 

Id: si 

Type: StorageResource 
Constraint: 

freeMb > 20000 
Ranking: 
freeMb 


Resource: 

Id: cl 

Type: ComputeResource 
Constraint 

freeCpus >128 

& freePhysicalMemoryMb > ‘TOG” 
& operatingSystem = ‘TRIX64 
Ranking: 
freeCpus 


Figure 2. Resource Broker request 


The global/local distinction for constraints is purely for 
notational convenience and natural encapsulation as local 
constraints are conjoined to the global constraint. Resource 
ranking, however, is always performed with respect to the 
global ranking function. Local ranking functions may only 
reference attributes of the associated resource and are used 
to prune resource sets during evaluation as described in sec- 
tion 3.4. If no global ranking function is defined, it is as- 
sumed to be the product of the local ranking functions. 

Constraints and ranking functions are written in typed 
first-order logic without quantification, which consists of 
constants, variables, functions, and boolean, arithmetic, 
and relational operators. The constants include numbers, 
strings, and the boolean values true and false. The variables 
consist of the set of resource identifiers given in the individ- 
ual requests and are used to reference resource attributes. 
In figure 2, the variables are “cl” and “si”, which are used 
in the global constraint to reference the host name of the 
compute and storage resource, respectively. 

Supported operators include standard boolean, arith- 
metic, and relational operators as well as “contains” and 
“! contains” for collection types such as sets and lists and 
the inline conditional operator ?: a la C and Java. All of 
the arithmetic, relational, and set operators are extensible 
based on the Number, Comparable, and Collection classes 
of Java, respectively. That is, any operands conforming 
to these interfaces may be used. For the relational opera- 
tors, if only one operand conforms to the Comparable inter- 
face, but its associated type has a constructor based on the 
other operand type, an appropriate instance will be created 
to make the two operands comparable to each other. This 
is especially useful for handling different unit types. For 
example, in figure 2, the “freePhysicalMemoryMb” func- 
tion returns a Comparable-conforming type “MemorySize”, 
which has both Number and String constructors allowing 
memory sizes such as “10G” to be easily normalized and 
compared to purely numeric values. 

The key element of the specification language is the 
set of functions available, which varies depending on the 
providers that have been incorporated into the system. 
Surfer has no built-in functions. Every function available 
is defined by a provider that has access to the information 
that function supplies. Figure 2 shows some of the functions 
available in the example resource broker discussed in sec- 
tion 4, which was developed using the Surfer framework. In 
the figure, “freeCpus”, “freePhysicalMemoryMb”, “operat- 
ingSystem”, “freeMb”, “host”, and “ranking” are all func- 
tions supplied by a specific provider. 

While Surfer requests are similar to the gangmatch clas- 
sad requests of [16], its pull-based model of information 
allows its language to be significantly more expressive. In- 
formation that is too large, complex, or dynamic to be pre- 
computed and pushed to a central matchmaker, can be easily 
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utilized by calling an appropriate provider function. 

The result of a Surfer request is a tuple set, which is a set 
of tuples such that each tuple is a map from variable names 
(i.e. resource ids) to resources that has been optimized for 
space and intersection performance. Figure 7 shows three 
example tuple sets. Each column shows a different variable 
value. For example, S2 has three tuples with variables vl 
and v2 where (vl, v2) takes the values (a, a), (b, a), and 
(b, b). The tuple set resulting from a request contains the 
specified number of tuples and is ordered by value of the 
global ranking function from highest to lowest. Each tuple 
contains a resource selection for each resource id requested 
and satisfies the conjunction of all specified constraints. 

3.2. Providers 

The core of the resource brokering framework are the 
providers that supply functions that may be constrained and 
queries that may be executed to produce resources with ap- 
propriate attribute values. Functions can be defined to take 
objects of any number and class type as arguments and to 
return objects of any class type. This allows significant flex- 
ibility and arbitrary extensions to the constraint and rank- 
ing language. Functions may also be macros, which are 
processed during rewriting and transform strings to strings. 
Macros provide an easy abstraction mechanism for hiding 
complexity and grouping commonly used expressions. 

Queries take a set of variable names and a boolean ex- 
pression utilizing those variables and return a tuple set of 
resources satisfying the expression. Providers are not re- 
quired to support queries, but query providers must support 
functions as the set of functions supplied by a provider de- 
fines the functions that are constrainable within the boolean 
expression of queries to that provider. 

The set of resource types that can be requested by the 
user is defined by the queries of the available providers. 
That is, any resource can be requested for which there ex- 
ists a query in some provider. Resources are considered to 
be objects with a set of identifying attributes that are unique 
to each resource (e.g. host and directory for a storage re- 
source) and a set of dynamic attributes that may vary over 
time (e.g. free disk space). Figure 3 shows the Java inter- 
face that every resource must implement. The getAttributes 
method must return the map from attribute names to values. 
The isMergeable method takes a resource and must return 
whether that resource has the same identifying attributes as 
the current resource. Finally, the mergeAttributes method 
takes a resource and merges its attributes into the current 
resource’s attributes. This method is used to merge the at- 
tributes of resources with the same identifying attributes 
that may have been produced by different providers during 
evaluation. A BaseResource class provides default imple- 
mentations for getAttributes and mergeAttributes. 


public interface Resource ( 

public Map getAttributes(); 

public boolean isMergeable(Resource res); 

public boolean mergeAttributes(Resource res); 


Figure 3. Resource interface 


Figure 4 shows the Java interface that every provider 
must implement, which consists of two methods for func- 
tions and two for queries. The getFunctions method must 
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hasQuery method takes a list of resource types and must re- 
turn a boolean indicating whether that type of query is sup- 
ported. The callFunction methods takes a function name 
and an array of arguments and must produce an actual value 
based on these arguments. The runQuery method takes a set 
of variable names and a boolean expression utilizing those 
variables and must return the resources satisfying that ex- 
pression. Section 3.5 describes how new providers are in- 
corporated into the framework. 


public interface Provider { 
public Set getFunctionsO; 
public boolean hasQuery(List types); 

one of / P ublic 0b ) ect callFunction(String name. Object!] args); 
\ public List callFunctions(List names, List argArrays); 

e of f public TupleSet runQuery (Set vars, AST ast); 

V public List runQueries(List varSets, List asts); 

1 


Figure 4. Provider interface 

For some providers, it may be possible to increase over- 
all throughput by processing function calls and queries 
in batches. This is especially likely in cases where the 
provider uses other services on different hosts to implement 
its back-end functionality. To avoid limiting any potential 
provider optimizations, providers can implement alternative 
batch interfaces for callFunction and runQuery. A Base- 
Provider class provides implementations of each in terms 
of the other, thus whichever is not provided will be based 
on the one that is. An additional optimization available to 
providers is to cache the dynamic attributes of any resources 
returned in a query. When this is done, functions based on 
those attributes can be called with minimal cost. 

3.3. Rewriting 

In order to produce an appropriate tuple set from a 
boolean constraint, that constraint must first be rewritten 
into an expression using the functions and queries that the 
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providers supply. Four set operations are utilized to de- 
scribe these expressions: intersections, unions, queries, and 
reductions. Intersections and unions are defined as normal 
but with special handling for tuple sets. A query takes a 
provider id, a set of variables, and a boolean expression in 
those variables and returns a tuple set such that each tuple 
has a resource mapping for every variable that together sat- 
isfy the boolean expression. A reduction takes a tuple set 
and a boolean expression and returns the subset of the given 
set that satisfies the boolean expression. In general, a query 
is more efficient than a reduction as each element in a re- 
duction set may necessitate a call to multiple providers. 

Rewriting begins with a request’s conjoined constraint, 
where conjunctions are changed to intersections, disjunc- 
tions to unions, and negations are distributed along the way. 
True constants are changed to universal sets (i.e. the set rep- 
resenting all possible resource combinations) and false con- 
stants to empty sets. Any macros are expanded by calling 
the appropriate provider function. Relations are changed 
to queries if all functions within the relation belong to the 
same provider and that provider supports queries in the req- 
uisite number and types of variables. All other relations are 
changed to reductions on the universal set, with each func- 
tion transformed into an explicit call to an explicit provider. 

Although the expression generated from this initial trans- 
formation could be evaluated as is, several optimizations are 
possible. In particular, it is desirable to ( 1) eliminate univer- 
sal sets as they are expensive to compute, (2) minimize the 
number of queries to run by combining queries to the same 
provider, and (3) minimize the number of function calls by 
minimizing the size of each reduction set. 

To achieve these goals, the unification axioms shown in 
figure 5 are applied to the initial set expression. Lower num- 
bered axioms are applied before higher numbered axioms. 
Axiom 1 actually consists of four separate axioms, the most 
important of which is the second, which will eliminate a 
universal set when it is intersected with anything else. Ax- 
ioms 2 and 3 are used to combine queries when possible. 
Axioms 4 and 5 are used to minimize reduction set size. 
Although axiom 5 can always be applied in place of axiom 
4, it is undesirable to choose the order in which the reduc- 
tions occur at this point since the number of elements in 
each set is unknown until they are actually evaluated. Fi- 
nally, axiom 6 is used both to eliminate universal sets and 
to minimize reduction set size by pushing intersections into 
unions to maximize the impact of the other axioms. The 
end result of rewriting will be an expression of the form 
Ui reduce* (flj query j, f\ k t k ). 

Figure 6 shows the conjoined constraint of figure 2 after 
rewriting to reduced set form. All of the macros have been 
expanded from an easily readable form to the specific names 
used by one of the providers. Note that “freeCpus” has been 
expanded to the value representing 100 times the number of 


1. An0 = 0 Anl = 4 A<J<b — A j4ui = l 

2. query(p, Vi,t\)r\query{p, V 2 , * 2 ) = queryip, Vi UV 2 , ti At 2 ) 
when p has queries in Vi U V 2 

3. queryip, Vi,ti)Uquery(p,V 2 ,t 2 ) = queryip, Vi UV 2 , ti Vt 2 ) 
when p has queries in V\ U V 2 

4. reduce(A,ti) n reduce(3,t 2 ) = reduce(A n B,ti At 2 ) 

5. An reduce(B, t) = reduce(A n B, t) 

6. An(BuC) = (AnB)u(AnC) 

Figure 5. Unification axioms 

free CPUs in the last minute divided by 100. This is an ex- 
ample of how complex details can be hidden from the user 
by supplying appropriate macros. All universal sets have 
been eliminated, queries have been combined as much as 
possible, and the one reduction set has been minimized by 
bringing all other terms inside. Note that the two remaining 
queries cannot be combined even though their providers are 
the same because that particular provider does not support 
queries of multiple types at once. 

reduce(query(pid2, {si }, $sl.Mds_Fs_freeMB > 10000) 
nquery(pid2, {cl}, 

$cl .Mds_Cpu_Total_Free_lminX100 / 100 > 128 
& $c 1 Mds_Memory_Ram _Total_frccMB > ”10C" 

& $cl.Mds_Os_name = "IR1X64"), 
call(pid2, Mds_HostJm, [$cl]) = call(pid2, Mds_Host_hn, [$sl])) 

Figure 6. Rewritten constraint 
3.4. Evaluation 

After a request has been rewritten into reduced set form, 
it must be evaluated to produce an actual tuple set, which 
must then be ordered according to the global ranking func- 
tion. This involves utilizing the functions and queries sup- 
plied by the providers and computing the results of set, 
arithmetic, and relational operations. Although the rewriter 
has performed expression-level optimizations, it is the re- 
sponsibility of the solver to optimize the actual evaluation 
of rewritten form. The main goals of optimization are to 
(1) utilize available provider optimizations, (2) run identi- 
cal queries only once, and (3) control set expansion. 

To take advantage of any available provider optimiza- 
tions, function calls and queries are always grouped into 
batches and sent to the batch interfaces of callFunction and 
runQuery in the providers. Identical queries may result 
from the application of axiom 5 in figure 5. To reduce 
the negative impact of this axiom, queries are gathered, ran 
once, and the results duplicated where necessary. 

Since Surfer allows an arbitrary number of resources to 
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be selected at once, the number of tuples that can be gen- 
erated during evaluation is exponential in the number of 
resources requested as each position within the tuple can 
potentially take any resource value of the appropriate type. 
Thus, the most critical optimization of the solver is to con- 
trol this complexity to the greatest extent possible. This 
exponential expansion occurs during intersections between 
tuple sets with disjoint variable spaces as each tuple in one 
set may generate a new tuple for each tuple of the other set. 
Unions do not have this problem as the new set size is guar- 
anteed to be at most the sum of the sizes of the two sets. 

Although the final size of an intersection involving mul- 
tiple sets is fixed, the sizes of the intermediate sets may vary 
with the order in which the individual intersections are eval- 
uated. The intermediate sizes affect the total number of tu- 
ples that must be compared, thus directly affect execution 
time. Figure 7 shows three tuple sets and the number of tu- 
ple intersections required for each evaluation order. As can 
be seen, even for small sets, the number of tuple intersec- 
tions is significantly impacted by the evaluation order. 


SI 

S2 

S3 


vl 

vl 

v2 

v2 

v3 

n order 

n’s 

a 

a 

a 

a 

a 

(Si n S2) n S3 

7 


b 

a 

a 

b 

(Si n S3) n S 2 

16 

b 

b 

b 

a 

(S2 n S3) n Si 

18 



b 

b 



Figure 7. Intersection order 

Surfer optimizes intersection evaluation order in two 
ways. The basic assumption of these optimizations is that 
sets will share variables that help reduce the resulting set 
size. When intersections occur outside of a reduction, the 
evaluation order is selected according to estimated set size. 
For two sets S 1 and S2, let the set of variables shared be- 
tween tuples of S 1 and S2 be denoted by V and the maxi- 
mum number of resources possible for each iq in V be de- 
noted by <7 i . The estimated set size of 51 D 52 is then de- 
fined as |S1| • |S2|/ nl=i a > ■ That is, for each tuple of SI, 
the number of tuples of S2 that can be expected to match 
that tuple and be added to the resulting set decreases by a 
factor of a, for each variable shared. Thus, sets that are 
very sparse will be intersected before denser sets and sets 
with more shared variables will be intersected before those 
with less. For simplicity, the implementation uses the same 
<7j for all resource types, which may be configured as de- 
scribed in section 3.5. 

When intersections occur within reductions, a different 
strategy is used. After rewriting, all reductions will be in the 
form reduce(f] q =1 Sj, /\£ =1 t k ). This form can be rewrit- 


ten to reduce(reduce(f] s i=l S,, ti ) nDy= s +i Sj,/\ r k=2 t k ) 
when no variable of term 1 1 is a variable of any set Sj . Thus, 
any intermediate result can be reduced by an individual term 
as long as that result contains all of the sets relevant to that 
term. In this case, the evaluation order is chosen based on 
terms. Terms are chosen in ascending number of variables 
and by involved estimated set size when the number of vari- 
ables is the same. The idea is that the fewer variables a term 
involves, the fewer sets will have to be intersected before 
they can be reduced. After a term is chosen, intersections 
are performed as in the non-reduction case. 

Figure 8 shows the time required to generate differ- 
ent numbers of resource tuples by intersection on a 750 
MHz Pentium 3 system with 256 MB of memory running 
FreeBSD. The curves show the three different methods by 
which the given number of tuples were generated based on 
an initial request for four resources with 120 possible val- 
ues for each. In the queries only case, each of the four re- 
source dimensions was reduced by the same factor using 
a constraint on each resource variable (e.g. Ssl.freeMb > 
10000). In the reductions only case, relations between pairs 
of variables were added to reduce the possible set size (e.g. 
Ssl.freeMb + $s2.freeMb > 10000). Finally, in the queries 
and reductions case, both techniques were used. 



Figure 8. Intersection time vs. # tuples 

Although the intersection times for even fairly large sets 
are reasonable, once the final tuple set is generated, each 
tuple must still be evaluated against the ranking function. 
If the resulting set is very large and the ranking function 
is complex, this evaluation may take significant time. To 
guard against this possibility, Surfer provides a configurable 
threshold that limits the maximum number of tuples that are 
evaluated when intersecting two sets. When more than the 
threshold tuple pairs are to be evaluated, a sampling func- 
tion is generated that selects threshold pairs of tuples from 
the two sets. Those tuples are intersected to produce the 
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resulting set, while the remaining tuples are discarded. 

With a threshold limitation comes a loss of precision as 
all possible resource combinations are no longer evaluated. 
There is also no guarantee how many tuples the resulting 
set will contain since providing such a guarantee could po- 
tentially require every tuple pair to be evaluated. In an at- 
tempt to keep the most desirable tuples. Surfer generates the 
sampling function based on the product of the local rank- 
ing functions associated with the resources of the tuple set. 
Each tuple set is ranked individually according to this func- 
tion and then the square root of the threshold number of the 
highest ranked tuples of each set are intersected. If no lo- 
cal ranking function is defined for a particular resource, a 
random sampling function is used. 

Once the final tuple set has been produced, the last step 
of evaluation is to order this set by the global ranking func- 
tion. First, the solver’s evaluation routines are used to com- 
pute a rank for each tuple. The tuple set is then sorted based 
on these values. Finally, the requested number of the high- 
est ranked tuples are returned to the user. Figure 9 shows the 
total evaluation time of the queries only case of figure 8 for 
different threshold values. As can be seen, the evaluation 
time is relatively constant once the threshold is reached al- 
lowing a maximum acceptable response time to be set based 
on the threshold, if desired. 



Total Resource 

Figure 9. Evaluation time vs. # tuples 
3.5. Implementation 

Surfer is implemented in Java as an Open Grid Ser- 
vices Infrastructure (OGSI) compliant service within the 
Open Grid Services Architecture (OGSA) framework [5], 
In the OGSA model, all grid functionality is provided by 
named grid services that are created dynamically upon 
request. The reference implementation of OGSI is the 
Globus Toolkit [3], which provides grid security through 


the Grid Security Infrastructure (GSI), low-level job man- 
agement through the Globus Resource Allocation Manager 
(GRAM), data transfer through the Grid File Transfer Pro- 
tocol (GridFTP), and resource/service information through 
the Monitoring and Discovery Service (MDS). 

Figure 1 shows the architecture of the Surfer broker- 
ing framework. In this figure, a client application uses the 
Surfer client API to request a given number of resources 
of specific types and characteristics. This request is con- 
verted to XML and transmitted to an instance of the Surfer 
service running within an OGSI container. The rewriter 
component is then used to rewrite the request into a sin- 
gle constraint and from there to the reduced set form of 
that constraint. This expression is then given to the solver 
component, which evaluates the given set. Any calls and 
queries are evaluated using the correlator component, which 
provides a common interface for accessing all the provider 
functionality. The solver also ranks the resource tuples 
based on the ranking function given by the client applica- 
tion. Finally, the resulting set is returned back to the client 
and converted from XML form back to a tuple set of re- 
sources meeting the specified characteristics. 

Extending Surfer from a framework into a usable re- 
source broker for a specific grid environment involves two 
steps. First, a set of providers must be implemented that 
conform to the Provider interface of section 3.2. These 
providers should reflect the information available in the 
given grid environment. The higher the quality of infor- 
mation supplied within the functions of these providers, 
the higher the quality of the selection results. Once the 
provider implementations are ready, the second step is mod- 
ifying Surfer’s Web Service Definition Language (WSDL) 
parameters. These parameters include the sampling thresh- 
old, the expected size measure cr*, and, most importantly, 
the providers parameter, which supplies the fully qualified 
class names of the provider implementations. The providers 
parameter is used by the correlator component to load all the 
available providers into the framework. 

After configuration. Surfer is available as a usable re- 
source broker. Users or client applications can request the 
types of resources that are selectable, the functions that are 
constrainable, or can make a request to return resources of 
the appropriate types with specific characteristics. 

4. IPG Resource Broker 

An initial prototype of a resource broker for the NASA 
IPG has been developed using the Surfer framework. Even- 
tually, this broker will have providers for most of the IPG 
services under development and will allow constraints on 
resource access based on Cardea, software locations, ver- 
sions, and dependencies based on the Naturalization Ser- 
vice, wait, execution, and transfer predictions based on 
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the Prediction Service, etc. For the initial prototype, two 
providers were implemented: an MDS provider and an 
MDS macro provider. The MDS provider supplies func- 
tions and queries based on the fields available in the MDS 
servers of the IPG. The MDS macro provider supplies 
macros to abstract the sometimes cryptic names of MDS 
into more human readable forms. 

The MDS provider supplies information about two types 
of resources: compute resources and storage resources. 
Compute resources consist of a host name, a queue name, 
and a queue type while storage resources consist of a host 
name and a directory name. Together, the two providers 
supply 80 functions with 65 of them coming from the MDS 
provider and 15 from the MDS macro provider. 

Figure 2 shows a request to the IPG Resource Bro- 
ker. This request uses only functions from the MDS macro 
provider, which are expanded to functions of the MDS 
provider as shown in the rewritten request of figure 6. This 
request examined 10,920 resource combinations and ran in 
15.7 seconds, the bulk of which was associated with query- 
ing six separate MDS servers. 

5. Conclusions and Future Work 

This paper has described Surfer, the Selection and 
Ranking Framework for Extracting Resources. Surfer al- 
lows new information providers and resource types to be 
easily integrated and allows arbitrary extensions to its con- 
straint and ranking language through provider functions and 
flexible evaluation of built-in arithmetic, relational, and set 
operators. Its pull-based model allows information sources 
too large and/or too complex for push-based models to be 
efficiently integrated into the resource selection process. By 
decomposing resource brokering into a framework indepen- 
dent of any specific grid environment, Surfer simplifies re- 
source broker development and can reduce duplication of 
effort across the grid community. 

There are a number of directions for future research. 
Several efficiency and accuracy improvements are possible 
in the evaluation routines of the solver. One inefficiency of 
the current implementation is that common terms in reduc- 
tions and ranking computations are recomputed for every 
tuple. These routines will be rewritten in the near term to 
compute such terms once, which should speed up evaluation 
considerably. Another optimization is to allow providers to 
give the cost of calling their functions, which may allow 
term reductions to be evaluated in a more efficient order. Fi- 
nally, the accuracy of estimated set size computations may 
be improved by using different heuristics or by allowing the 
expected size measure <Ji to be configurable for each re- 
source type or be automatically derived during evaluation. 

Another area requiring more study is handling conflicts 
between providers such as different providers that supply 


the same information about the same resources or supply the 
same information, but about different subsets of resources. 
The former case could potentially be handled by allowing 
providers to supply a freshness measure to their information 
while an aggregate provider class may handle the latter. 

Additional functionality may be added to the specifi- 
cation language such as additional string operations like 
startsWith or endsWith. Other operators may be added as 
necessary. Finally, The EPG resource broker will be greatly 
enhanced in the near future. Namely, a variety of new infor- 
mation providers will be integrated into the system as they 
become available as part of the IPG project. 
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